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ABSTRACT 

The goals of this research were to: (1) develop 
instruments useful in profiling the characteristics of teaching 
manuals used with grade 4 science textbooks and the effects of those 
manuals on teacher and pupil activities; (2) look at both high and 
low inference perspectives in making these profiles; (3) develop two 
instruments, one for elementary classroom teachers and the other for 
their pupils; and (4) use samples from these science education 
populations plus the judgments of science education professionals to 
profile two commonly used science teacher's manuals. Teaching manuals 
were analyzed because they are thought to be especially influential 
in elementary classrooms where reading is still not a fully developed 
skill. Since they are often the source for process-product/inquiry 
activities, teacher's manuals often play a central role in 
determining the quantity and quality of hands-on experiences students 
have. The instruments were administered to 30 fourth-grade teachers 
in five states who use the Silver Burdett, Merrill, or Scott Foresman 
text series. Seventeen teachers returned their answers as well as 
those of their 360 students. The graphical comparisons of teacher 
responses show both similarities and differences between the various 
teacher's manuals. Teacher and student survey forms are appended. 
(MVL) 
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Purpose 

The goals of this research were to: 

1 . develop instruments useful In profiling the characteristics of teaching manuals used 
with grade 4 science textbooks and the effects of those manuals on teacher and pupil activities, 

2. look at both high and low inference perspectives in making these profiles, 

3. develop two instruments, one for elementary classroom teachers and the other for 
their pupils, 

4. use samples from these science education populations plus the judgments of science 
education professionals to profile two commonly used sdence teacher's manuals, 

5Mm 

The Criteria for the Analysis and Selection of Science Texts Project ( CASST Project) was 
initiated in 1985 to focus the efforts of science educators on the improvement of the quality of 
science textbooks used in American schools. As a part of their work for CASST, Good and 
Shymansky (1986) called for research to help establish criteria for pre-high school science 
textbooks. They discussed the major status of the science textbook as the primary curricular 
element in science education, and called for answers to two major questions: 

1 . What Is the nature of the science content that 
should appear In science textbooks? 

2. How should this content be presented so as to optimize 
comprehension by students? (p. 12) 

This paper seeks to address each of these questions. It assumes that the nature of science 
content and the "how" of presentation are best determined by judgments and research findings o; 
professional science educators. Their contributions to the instmments developed by this study 
are discussed later. 

To begin their work for CASST, Good (1985) surveyed the fifty states to assess the text 
selection criteria used in each. Among his conclusions and recommendations from that study 
were the following: 

1, At least two levels of appraisal forms should be developed. (A higher level form 
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assuming more background information for proper use and a lower fonn not requiring the same 
level of expertise.) 

2. Appraisal fonns should be both science content specific and directed at grade level 
groupings such as K-3, 4-6, etc. 

3. There should be much more emphasis on problem solving in science. 

4. There should be more emphasis, in the elementary grades, on lab-based development of 
science process skills and related knowledge of concepts. 

5. Much more emphasis at the elementary grades should be placed on appraisal of teacher's 
manuals and supplementary materials. 

Goorfs first recommendation, calling for bo;h high and low inference analyses is based on 
the unique strengths of each type of observer. High inference observers can determine the extent 
to which science teaching materials meet complex goals that may be only partially seen and 
understood by others. Low Inference analyses rely on intimate knowledge of materials content 
that is gained most easily from experience with the material. Good dtes Howe (1985) in 
concluding that text appraisal instmments should combine the virtues of both high-inference 
and low-inference analyses. For this study, Meyer, Cmmmey and Greer's (1988) careful 
analyses of elementary teachers manuals were used for a high inference look at these materials. 
Low inference items, developed by the author, were used to develop the elementary teacher and 
pupil instruments, 

A look at current text appraisal instruments from many states, convinced Good that 
because of the different nature of pupils and the learning environments among the different 
grade levels, separate instruments should be developed to analyze various grades of instructional 
materials. If pupils themselves are to be a part of this analysis, as this study proposed to do, 
then the design of different instruments for each grade level was a necessity. Therefore for the 
purposes of this study, only fourth grade science teacher's manuals v^/ere examined. 

Good (1985) also found from his review of the state instruments that problem solving in 
science and lab-based development of science process skills needed more emphasis via heavier 
weighting in the appraisal fonns. The instruments developed for this study include a number of 
items assessing these issues. 

Meyer, Crummey and Greer (1988) recently analyzed elementary science texts and 
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teaching manuals to determine their general content and attributes and to make comparisons 
across publishers. In the teaching manuals, they found differences In the number of 
lecture/discussion and hands-on activities. Their analysis concluded that Merrill had ^..far 
more lecture/discussion activities and far fewer teacher-directed hands-on activities than 
Silver Burdett." Their analysis also found more activities in the Silver Burdett series. Table I 
shows the results of their analysis for the fourth grade teacher's manuals considered by this 
study. In their discussion, they suggested that science achievement needs to be considered in 
relation to actual teacher use of these materials as well as their contents. 



TABLE I 

HOW INFORMATION IS PRESENTED FROM THE TEACHER 
PRESENTATION BOOK AND'STUDE^fT MATERIALS IN THE SILVER BURDETT 
AND MERRILL SCIENCE PROGRAMS (from Meyer, Crummey, and Greer (1988)) 



Number of Various Activities in Teacher's Editions 
Program Grade . 

Lecture/ Hands-On 
Discussion Activities 



Silver Burdett 4 72 16 

Merrill 4 82 2 



The study reported here sought to address these highly influential teacher's manuals which 
accompany most elementary science texts because they are thought to be especially influential in 
elementary classrooms where reading is still not a fully developed skill. Good and Shymansky 
(1986) make the point that with the importance science educators give to 
process-product/inquiry, ancillary materials, which are often the source for such activities, 
can play a central role in detennining the quantity and quality of "hands-on" experiences 
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Students have. 

Instrument Devetopment 

Based on Goorfs 1985 report and on work at the Center for the Study of Reading, 
Champagne, Illinois, Jane Armstrong, one of the CASST Advisory Board members, developed a set 
of criteria for use in evaluating science textbooks. She suggested analysis be done in the areas of 
instructional design and organization, instructional strategies, the process of inquiry, content 
and using science. Since her suggestions are an excellent starting place and parts of these same 
topics are relevant to the quality of supplementary teaching manuals, many of her suggestions 
were used as guidelines in developing the instruments for this study. Table II details the items in 
the teacher's instrument and indicates the content of each item along with the researcher upon 
whose work the content was based. For example, Mayer's name next to items #1 0 and #15 
s. jws his work to be the source for those content ideas. Other science educator researchers are 
identified in Table II and a relevant, if any, publication is cited. 



TABLE I! 

Analysis of Instruments bv Contf^nt Qri pin and InffirencQ Level 



Content Category 



Content Origin 



Teacher's Instrument 
Item Item Inference 
Letter Number Level 



Addressed by 
EUEillS Instru- 
ment? 



inquirv/Discoverv/Process 












Frequency of discovery 


Mayer (1983) 


F 


10 


Low 




Promote process skills? 


Armstrong 


D 


8 


Low 




Test items measure process? 


Armstrong 


Ma,b 


20 


Low 




Concrete/Hands-on 












Percentage of Hands-on 


Armstrong 


1 


13 


Low 


yes 


Familiar and concrete under- 










pinnings suggested? 


Mayer (1983) 




15 


Low 


yes 


Quantity of activities 


Bredderman 










(1983) 


H 


12 


Low 


yes 


Common background experiences Finley (1983) 


K 


16 


Low 


yes 


Science applied to daily lives? 


Armstrong 


E 


9 


Low 


yes 


Reading. Vocabularv and Science 












Importance of Reading the text 


Shymansky and 










(Reading vrs. Activities) 


Yore (1979) 


Ca,b,c 


7 


Low 
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Help for below-grada readers? 


Shymansky and 










Yore (1979) 


Ga^bjCfd 


11 


Low 


Importance of science words 


Armstrong 


Ma,b,c 


20 


Low 


tieiDTUiness oi manual 










boid-tacea Headings 


Armstrong 


A 


5 


Low 


Accuracy of procedures 


Armstrong 


J 


14 


Low 


Suggestions actually used 


Evans 


B 


6 


Low 


Science background for teacher? 


Armstrong 


L 


19 


Low 


Evaluation Sugpastlon^ 










Evaluation ideas during chapter? 


Armstrong 


o 


22 


Low 


Does evaluation match lessons? 


Armstrong 


P 


23 


Low 


Kinds of evaluation suggestions 


Armstrong 


N 


21 


Low 


Cognitive Skills 










Promote cognitive conflict? 


Posner et al 










(1982) 




17 


Low 


Promote higher thinking skills? 


Armstrong 




18 


Low 


Validity of this instrument 


Evans 


R 


25 


High 


Affective 


Evans 


(not asked) 



yes 



One reason this study looked at the judgments of science education professionals like 
Meyer, Crummey and Greer (1988) as well as at classroom teachers was to take advantage of 
their respective abilities to provide high and tow inferential analyses of teaching materials. 
Classroom teachers who have just taught a chapter or unit using a teacher's manual are in an 
excellent position to quantify, tabulate and generally identify the characteristics of that manual. 
Professionals in science education are able to make complex judgments about a manual's nature 
which may be beyond the sophistication of tt ^ typical teacher. 

The pupil perspective is also helpful in assessing the characteristics of a parttoular 
teacher's manual. If the manual encourages one sort of teacher activity and the students 
consistently report another, then that particular manual's ability to facilitate instructional 
methods would obviously be questioned. Since elementary pupils have limited abilities to 
respond to written questfonnaires, their questionnaire contains only four items. Two seek the 
kinds of activities and learning the students recall from a recently completed chapter or unit. 
Another question assesses their feelings about the lessons (i.e. boring, interesting, too long, too 
short, fun, hard, etc.) and the last attempts to measure their success as science students. 
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A "pilor trial of these instruments plus a parallel one developed for administration to 
science education professionals was conducted to: judge the value of using instruments on 
different populations to analyze ancillary science teaching materials; determine the extent to 
which the instruments coincide and complement each other in their assessments of teacher's 
manuals; assess the validity of the instruments from the point of view of the populations using 
them and to determine the practical problems in administering these instruments. Results of 
this pilot study were reported at the NARST meeting in Washington in 1987, Evans (1987). 

A rather consistent "profile'' among the teachers surveyed, showed that that instrument 
does provide one useful method of desv':ribing the characteristics of teacher's manuals. However, 
since the pilot instalments were only used with one publisher's materials it was not known if 
the ''profiles'' of other manuals would have been significantly different. 

The pupil's responses, while unable to contribute large amounts of information to a 
teaching manual's profile, did add two important dimensions. Regardless of what classroom 
teachers using a manual and sdence educators reading it say about the kinds of science activities 
it promotes, a class of students independently reporting on what they actually did during a unit, 
is an excellent source of information about the relative success of different teaching manuals in 
promoting science process activities. In addition, the ability to access the affective responses of 
students to a particular chapter, taught with a certain teaching manual, provides an important 
dimension to an analysis which is difficult to obtain from any other source. 

Methods 

This study built upon the pilot by revising each of the instruments. New shorter versions 
were designed to have greater scaling uniformity, and an attempt was made to eliminate useless, 
redundant and non-discriminating questions. The teacher's questionnaire is included in this 
report as Appendix A and the pupil's as Appendix B. Last spring these new instruments were 
given to thirty fourth grade teachers who use the Silver Burdett, Merrill or Scott Foresman text 
series in North Carolina, Minnesota, Texas, Missouri and Ohio. Seventeen teachers retumed 
their answers along with those of their 360 pupils. Teachers were asked to report on any 
chapter or unit they had just finished. This variety of content was encouraged both to give the 
resultant profiles greater validity for the entire series and to make the test for differences more 



vigorous. Gr^hical analysis of tlie results as weii as Mests on differences between various 
means were produced. 

BfiSUlla 

The means for each scoreable item on the teacher^s questionnaire are presented for the 
five classes using the Mem'll text and the twelve teachers using Silver Burdett in Figure 1 . 
Since only one teacher and dass in the sample used Scott, Foresman materials, that series is not 
included in this report. Most responses showed no significant differences in teacher perceptions 
of the two ma..jals. However, question Ca (#7a), which asked teachers to report how activity or 
reading oriented they thought the first teaching suggestion in their manual was, found Silver 
Burdett significantly (at the .02 level) more activity oriented than Merrill. Questions Ga and Gd 
(#1 1 a and 1 1d) found that the Silver Burdett manual gave significantly greater (at the .04 
level) help to teachers in wori<ing with below-grade level readers than did the Merrill manual 
and, while not significant, questions Gb and Gc on the same topic were also more positive for 
Silver Burdett. The other significant difference was in question L(#19) which discovered a 
teacher perception of significantly greater (at the .04 level) background information in the 
Men^ill manual than in Silver Burdett's. 

Although not significant, question D (#8) results showed that teachers using Silver 
Burdett reported more science processes actually used by them in teaching than did teachers 
using Merrill. Another non-significant result for question F (#10) found Merrill teachers 
reporting more "discovery*" learning activities in their manual than Silver Burdett teachers. 

Figure 2 graphically compares the means for the two major pupil questions. Unpaired 
t-Tesls for significant differences between these means are reported in Tables III and IV. The 
first question (see Appendix B), which asked students to name all the activities they did during 
the unit, found significantly more activities reported by Silver Burdett students than pupils 
using Merrill materials. The second question, which asked students to list all the content things 
they learned about, also found a significantly greater number among the students exposed to 
Silver Burdett. Their attitudes towards science were not significantly different. 

Discussion 

Even with this small sample of teacher opinion, the profiles of the Merrill and Silver 
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COMPARISON OF MEANS OF PUPIL REPORTS 




■ Merrill 

H Silver Burdett 



Activities Content 

FIGURE 2 
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TABLE ill 

UNPAIRED t-TEST: SERIES MANUAL VRS. CONTENT 



J2El Unpaired t Value: £rabJ2it2i!i: ■ 

359 -3.908 .0001 

Oroup: Count Mean: Std.Dev.: Std. Error: 

Merrill 79 3.241 2.879 .324 

Silver Burdett 282 4.979 3.646 .217 



TABLE IV 

UNPAIRED t-TEST: SERIES MANUAL VRS. ACTIVITIES 



m Unpaired t Value: Prob. (2-tain: 

359 -4.646 .0001 

Group: Count: Mean: Std.Dev.: ' ' 'd. Error: 

Merrill 79 .772 .933 .106 

Silver Burdett 282 1.589 1.481 .088 
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Burdett teacher's manuals shew clear, and in the case of 17% of the items, significant 
differences. These teacher's opinions coincide with Meyer, Crummey, and Greer's (1988) 
findings that Silver Burdett has more hands-on activities and Memll has more 
lecture/discussion suggestions. This stud/s teachers found tha first Silver Burdett suggestions 
in various units to be significantly more activity rather than reading oriented. On the other hand 
they found that Menill provided significantly more background information for the teacher, 
which can be assumed to be espedally useful to teachers engaged in lecture/discussioFi activities. 
The consistently greater help with below-grade level readers which teachers found in the Silver 
Burdett series was not reported in the Meyer, Crummey, and Greer (1988) study. These unique 
profiles by teachers who have actually taught the units they are describing, can provide 
important insights for sdence educators into the perceived attributes of teacher's manuals. Are 
manual attributes, which are not recognized as such by teacher users, useful components or do 
they need to be revised? 

Since the pupil reports of both significantly greater content learning and number of 
activities engaged in for Silver Burdett as compared to Memll are the end results of textbook 
and teacher manual usage, ^;hey are especially important to science educators. This is 
particularly true since the Meyer, Crummey, and Greer (1988) study found more activities in 
the Silver Burdett teacher's manuals and this study's teachers found some evidence of at least the 
first manual suggestions being more activity oriented. The possibility, advocated by Good and 
Shymansky (1986), that teacher's manuals do indeed have an influence on the quantity of 
science aciivities is enhanced. 

Further work should include a larger sampling of teachers and of the other elementary 
text series so that strongly representative profiles can be created for each manual. The 
usefulness of pupil reports should be further exploited with other questions intended to assess 
outcomes of series use. It would also be helpful to ask teachers more questions about their actual 
teaching strategies resulting from their use of the teacher manuals. 
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SURVEY OF TEACHER'S MANUALS FOR 

ELEMENTARY SCIENCE TEXTS 
Thank you for agreeing to complete tMs questionnaire about the teacher's manual you used for a recent 
science chapter. Please base all of your answer on the manual suggestions for tho chapter you taught, not c 
other chapters in the manual nor on other manuals. You'll probably need the teacher's manual for reference 
while filling out this questionnaire. 

1. What is the name of the teacher's manual about which you're reporting? 

2. Who is the publisher of this teacher's manual? 

3. What date was it published? 

4. What is the name of the chapter about which you're reporting? 

A 5. Review the bold faced headings in the manual for the chapter you taught. When you taught this materia 
were you able to locate the teaching ideas and assistance you needed just by using these headings? 

I I I I I I I 

always sometimes never 

B 6. Look through the teaching suggestions for the chapter you used in the teacher's edition. Approximately ^ 
many of the manual's suggestions for this chapter did you actually use? 

7.Ust the first 3 teaching suggestions, for the chapter you're reporting on, in the spaces below. Then mark ale 
the answer bar how much reading is associated with each suggestion. 

CA 1st Suggestion: 



only involves reading a mixture only involves an activity 
CB 2nd Suggestion: 



only Involves reading a mixture only involves an activity 



CC 3rd Suggestion: 
I I I- 



only involves reading a mixture only involves an activity 

D 8. Now, use this list to tell which of the science processes is encouraged by the activities you used from th 
manual. Circle all that you used with children as a result of the ideas contained in this manual. 

1 observing 6 inferring 

2 experimenting 7 analyzing 

3 verifying 8 synthesizing 

4 predicting 9 generalizing 

5 organizing 

E 9. Count the number of times in this chapter of the teacher's manual where ideas are given for applying scii 
to children's everyday lives. 

F 10. How many times does the teacher's manual suggest "discovery" learning activities where students can 



V basically dtecover for themselves the scientific concept under consideration? 

1 1. Based on the chapter youVe taught, indicate whether or not the teacher's manual provides suggestions 111 
the following to he!p below-grade level readers. 

GA. .how to rearrange topics or chapters into a sequence which is more appropriate for the students' reading 

I I I I I L_l 

a fot some none 

GB. .how to use conceptually similar material from other sources that is more appropriate to the students* 
reading level 

I \ I I \ I I 

a tot some none 

GC. how to develop study guides for difficult reading sections 

I I I L I I I 

a tot some none 

GD. .how to make verbal tapes of difficult reading material for less able readers 

I I I I I 1 I 

a tot some none 

H 1 2. How many different activities for students are suggested for the chapter you taught using the teacher's 
manual? 



1 13. From all the suggesttons given, what percentage are for hands-on activities for students? 



0% 20% 40% 60% 80% 1 00% 

J 14. From the suggestions for this chapter, recall the apparent inaccuracies in either content or in the 
step-by-step procedures. Hew many instances did you find in this one chapter? 

15. Rnd an instance where the teacher's manual gives a suggestion for providing students with familiar and 
concrete underpinnings of scientific ideas before introdtj cino new ideas . (For example, if you were studying 
energy, the manual might suggest that you have your students rub their hands together until they are warm.) 
Describe that suggestion briefly • 



How many times does the manual provide suggestions like the one youVe described, in the 
chapter you taught? 

K 1 6. Check the manual to see whether common classroom background experiences are suggested for studt 
do BEFORE they are asked to read a textual passage. What percentage of lessons provide this common exp 

I I I I I I 

0% 20% 40% 60% 80% 100% 

17. How many activities suggested by the teacher's manual promote cognitive conflict in the students. That is 
how many activities reveal something about the world that goes against what the students think before they c 

. the activity? 

1 8. Look at the enrichnnent suggestions for the unit or chapter you are reporting on and tally each activity int 
one of the following categories: 

flower level thinking called for (knowledge, understanding) 

^higher level thinking needed (analyzing, synthesizing, evaluating) 

ERLC ^ « 



teacherr "^^"^ P^^^S^^P^^ of background information does this chapter of the teacher's manual provide for 

20. Review ail the evaluation quesfions suggested by the teacher's manual. Estimate the percent which ate 
devoted to measuring each of the following: *^ 
MA learning scientific vocabulary 
■ J 



0% 20% 40% 60% 80% 100% 

MC learning from hands-on activities 

I 1 1^ I I I 

0% 20% 407o 60% 80% 100% 

learning scientific concepts 



0% 20% 40% 60% 80% 100% 

N 21 Circle each of the following evaluation methods suggested by the teacher's manual for your selected 
cnapter: ' 

a. knowledge test questions h. true and false questions 

b. comprehension test questfons j. short answer questfons 

c. questfons requiring students to apply knowledge j. matchina questions 

d. higher-level test questions k. paragraph essay questions 
f • student-self tests I. filNn-the-blank questions 

t. tests involvmg manipulation of objects m. questions with drawings 

g. multiple-choice questfons n. questfons using matJiematics 

as aUh^end?^ teacher's manual provide suggestfons for evaluating student progress during the chapter as 

P 23 To what extent do the evaluation questions mateh the science infonnation covered in this Chapter*? The 
are the questions about exactly the same science ideas covered and in the same amount' 

L— I 1 i 1 I I 

exactly somewhat not at all 

24. Rnally, counting the current year, how many years have you taught school? 



R 25. To what extent do you think this questionnaire allowed you to tell us about the contents of the teacher's 

mailUal • 



J 



completely partially hardly at all 



THANKS AGAIN FOR YOUR TIME AND PROFESSIONAL ASSISTANCE! 

Please use the stamped manila envelope 

to return this questionnaire and your 
student's questionnaires to: 
Robert H. Evans 
Education Department 
Wake Forest University 
Winston-Salem, N.C. 27109 
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Write a letter to your friend telling them everything you've 
done In science with recently. 

(Your teacher will tell you the science subject to write about.) 

Dear , 

/ Va done all of these things In science while we 
studied about 



Please turn this paper over and do the other side. 



Tell your friend about studying . 

Circle the words below which tell what It was like. 



boring Interesting good bad fun hard 
too long too short just about right 



Your friend wants to know the most Important things you learned 
while studying . Write them below. 

The most important things I learned were.... 



Did your teacher give you a test or quiz on ? 

Yes or No If "yes", how did you do? (circle one answer) 

very well pretty good OK not very good very poorly 



